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DETAILED ACTION 



Response to Amendment 



1. 



This office action is in response to an amendment filed 1 1/10/2005. 



2. 



Claims 1 and 37 have been amended. 



3. 



Claims 15 and 40 have been cancelled. 



Claim Rejections - 35 (JSC § 103 



1. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

2. Claims 1-14, 16-39, and 41-47 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Guell et al. (NPL Document "FLILO - an Enhanced Vision System", 
herein referred to as "Guell") in view of NASA et al. (NPL Document "NASA's High- 
Speed Research Program", herein referred to as "NASA") in further view of Bernier et 
al. (US Pub 2004/0169663, herein referred to as "Bernier"). 

As per claim 1 , Guell teaches the claimed "first sensor" and teaches the claimed 
"second sensor" by teaching of a plurality of imaging sensors shown on pg. 34, figure 4. 

Guell teaches the claimed "fuse the images to a single viewpoint" by teaching of 
"The video images from the cameras are digitized, and electronically combined in the 
Electronic Imaging System (EIS) Processor to form an external view to the pilot" (pg. 33, 
1st full paragraph in 1st col). 
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Guell teaches the claimed "transform the fused image to a first viewpoint image 
... and a second viewpoint" by teaching of in figure 6 where a cockpit is shown with 
displayed titled images. If a pilot and co-pilot were both sitting in the cockpit, each pilot 
(operator) would have a different viewpoint image from their respective positions within 
the cockpit where the set of tiled images in constructed from fused images from multiple 
sensors. Further, Guell teaches the use of having multiple pilots in the cockpit by 
teaching of "FLILO would help retain the two-man cockpit crew" (pg. 34, last of line of 
1st col and first line of 2nd col). Further, Guell teaches the claimed limitation by 
teaching of "Each pilot has an independent view, which is controlled by their head 
position, while utilizing the same sensors that are static and fixed to the aircraft" (pg. 32, 
middle of 1st col). 

Guell does not explicitly teach the claimed "detect moving objects in the images". 
NASA teaches the claimed limitation by teaching of a feature described as "Object 
Detection: Data was gathered to help develop methods for detecting other aircraft" (third 
item in bulleted list under section "XVS Flight Test Series II"). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to combine Guell and NASA. One advantage of the combination is provided 
by NASA, which teaches of "In addition to providing valuable real-time sensor data for 
subsequent analyses, the first XVS flight test series gave researches confidence that a 
future supersonic passenger jet could indeed be flown without forward facing windows 
in the cockpit" (3rd paragraph under the section "XVS Flight Test Series I") where the 
functionality of moving object detection would aid in achieving a windowless cockpit. 
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Guell does not explicitly teach the claimed "to generate a common display area 
associated with at least two mutually exclusive windows of information on the display 
device, wherein the common display area can be customized by the operator to display 
detailed information related to the information displayed in the associated windows". 
Bernier teaches the claimed limitation by teaching of in figure 16 a common window, 
124 is generate from two separate image sources 122a and 122b (windows of 
information). Bernier in figure 1 8 shows that the common window can have mutually 
exclusive (not overlapping) windows of information, 134 and 136 and that the common 
window is customizable where the added windows of information 134 and 136 are 
customized added features. 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to combine Bernier with the combinable system of Guell and NASA. One 
advantage of the combination is provided by Bernier, which teaches of "the image tiling 
capability also provides the ability to present picture-in-picture virtual displays 
distributed throughout the viewable space. For instance, it may be desirable to have a 
moving map presented in the lower display areas ... Another desire may be to have a 
rear-facing camera mapped to a tile in the upper display area, similar to a rear-view 
mirror ... This imagery can be further augmented with a synthetic image source, such as 
a head-tracked 30 terrain rendering correlated with vehicle position" (paragraph [0100]). 
These common display area features and the option to pick and choose one or more of 
these picture-in-picture displays helps the pilot of the system of Guell and NASA get a 
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stronger idea and understanding of the overall surroundings of the aircraft while its in 
flight. 

As per claim 2, Guell does not explicitly teach the claimed "combine the first and 
second viewpoint images with symbols". NASA teaches the claimed limitation by 
teaching of "pilot's ability to control and land the aircraft relying only on sensors and 
computer-generated images (including various symbols) on the XVS display" (3rd 
paragraph under the section titled "XVS Flight Test Series I"). 

As per claim 3, Guell does not explicitly teach the claimed "detecting moving 
objects ... are to configured to execute simultaneously". NASA teaches the claimed 
limitation by teaching of multiple sensors (third paragraph under section "XVS Flight 
Test Series II") where these sensors would both be utilized to detect moving objects. 

As per claim 4, Guell teaches the claimed "transforming the fused image to the 
first viewpoint image" and teaches the claimed "transforming the fused image to the 
second viewpoint image" by teaching of on pg. 32, figure 1 , where the plurality of 
imaging sensors data is inputted into an "Electronic Imaging System (EIS) processor" 
and the resulting image data is then sent to a pilot for display. While Guell does not 
explicitly teach the claimed use of a plurality of processors, it would have been obvious 
to use a plurality of processors for real-time data processing in order to send the image 
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data to more than one operator (different viewpoints) because it is well known in the art 
that multi-processor use can be much faster. 

As per claim 5, Guell does not explicitly teach the claimed "symbols represent the 
moving objects in the vicinity of the device 1 '. However, it would have been obvious to 
one of ordinary skill in the art at the time of invention to modify Guell and NASA to 
perform the claimed limitation because Guell teaches of symbols in figure 6A and NASA 
teaches of detecting moving objects (third item in bulleted list under section "XVS Flight 
Test Series II"). The modification can be achieved by extending the moving objects 
detection capabilities of NASA to be represented as symbols such as the symbols 
mentioned by NASA where NASA specifically teaches of "computer-generated images 
(including various symbols) on the XVS display" (3rd paragraph under the section titled 
"XVS Flight Test Series I"). One advantage to the modification is that displaying 
symbols to represent moving objects around the aircraft (i.e. other aircraft) is one of the 
most convenient and useful data objects to display when navigating an aircraft. 

As per claim 6, Guell does not explicitly teach the claimed "wherein at least one 
of the first and second viewpoint images include environmental information for the area 
where the device is operating". Bernier teaches the claimed limitation by teaching of 
"For example, the aircrafts HSI and ADI displays, altimeters, airspeed, etc. could be 
displayed" (paragraph [0102]). Further, Bernier teaches of a moving map tile 
(environment information) in figure 18, piece 136. 
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One advantage of using the moving map of Bernier with the system of Guell and 
NASA is that Bernier teaches of "For instance, it may be desirable to have a moving 
map presented in the lower display areas, similar to having a paper map in your lap" 
(paragraph [0098]). Thus, this map feature makes it easy on the pilot to view the map 
information and prevents the pilot from having to look at a separate physical map on the 
pilot's lap while using the aircraft display system. 

As per claim 7, the reasons and rationale for the rejection of claim 5 are 
incorporated herein. Guell does not explicitly teach the claimed "wherein the symbols 
represent weather hazards in the vicinity of the device". However, it would have been 
obvious to one of ordinary skill in the art at the time of invention to modify Guell, NASA, 
and Bernier to perform the claimed limitation because Bernier teaches of "Visual 
hindrances may be due to bad weather, such as fog, snow, or rain, or they may be due 
to the time of day, such as night, dawn, or dusk. Further, some visual hindrances are 
due to the field of view limitations of the aircraft itself. Many aircraft cockpits have a field 
of view that is typically limited to a forward facing area that does not provide the pilot 
with adequate visualization to the sides and rear of the aircraft" (paragraph [0005]). The 
modification can be achieved by taking symbols such as those taught by NASA and 
using them to represent the weather data of Bernier. One advantage to the modification 
is that weather hazards may be difficult to see exactly at night and such symbols would 
help the pilot understand the aircraft's surroundings better. 
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As per claim 8, Guell teaches the claimed "to receive an enhanced image from a 
third sensor configured to provide an image of the out-the-window scenery in low- 
visibility conditions" by teaching of 'Typical sensors used with the system are low-light 
video cameras, long-wave 'infrared sensors, and millimeter wave radar, to name a few" 
(paragraph [0050]). 

As per claims 9 and 10, Guell does not explicitly teach the claimed "fuse the 
single viewpoint image with the enhanced image" and does not explicitly teach the 
claimed "utilize data from at least one position sensor to determine the location of the 
objects with respect to the device". Bernier teaches the claimed limitations by teaching 
of "For example, given a visible and an infrared image covering similar fields of view, 
the images can be combined at the pixel level, where priority can be given to the 
infrared image based upon its pixel intensity" (paragraph [0017]). 

Bernier teaches one advantage to combining the visible and infrared image 
features of Bernier with the system of Guell and NASA by teaching of "For example, one 
tile image may be defined with a 30% intensity and the other a 70% intensity. The 
images are then summed together in this intensity percentage ratio, thereby providing 
the user with the benefit of both images" (paragraph [0017]). Thus, the pilot using the 
imaging system of Guell and NASA would benefit from receiving a blended and 
effectively combined image display, because a blended image provides information of 
two sources simultaneously to aid with accurate imaging (i.e. visible and infrared image 
data coming from external sensors located on the outside of the airplane). 
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As per claim 1 1 , Guell does not explicitly teach the claimed "utilize data from off- 
board data sources regarding the objects" Bernier teaches the claimed limitation by 
teaching of "the central processor receives the output of a synthetic vision system that 
generates 3D terrain and graphics from a navigation database" (paragraph [0101]) 
where this database would contain off-broad 3D terrain data and off-broad navigation 
data since the aircraft would have limited capabilities of obtaining ground data through 
on-board ground sensors if at high altitudes. 

As per claim 12, Guell teaches the claimed "wherein the first sensor and the 
second sensor are video cameras" by teaching of "video images from the cameras are 
digitized, and electronically combined in the Electronic Imaging System (EIS)" (pg. 33, 
first full paragraph in first column). 

As per claim 13, Guell teaches the claimed "wherein the third sensor is a 
RADAR" by teaching of 'Typical sensors used with the system are low-light video 
cameras, long-wave 'infrared sensors, and millimeter wave radar, to name a few" 
(paragraph [0050]). 

As per claim 14, Guell teaches the claimed "wherein the third sensor is a FLIR 
sensor" by teaching of 'Typical sensors used with the system are low-light video 
cameras, long-wave 'infrared sensors, and millimeter wave radar, to name a few" 
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(paragraph [0050]), where the long-wave infrared sensors perform a similar function to 
the FLIR. 

As per claims 16, 24, and 37, the reasons and rationale for the rejection of claim 
1 are incorporated herein. Guell does not explicitly teach the claimed "output the first 
operator viewpoint image ... wherein the display device are positioned to provide to 
provide the portion of a desired out-the-window visual scene in combination with a 
cockpit window that provides another portion of the desired out-the-window visual 
scene". Bernier teaches the claimed limitation by teaching of "Importantly, also 
connected to the image processor is a display 28. In this particular embodiment, the 
display is a helmet-mounted display located in the helmet of the pilot or other viewer" 
(paragraph [0052]) and by teaching of "It is understood here that a helmet-tracking 
device is not required for the invention" (paragraph [0052]) where the reference would 
achieve the claimed limitations by using a monitor instead of a head mounted display (a 
different embodiment) where the monitor is placed in the cockpit (the cockpit with 
windows is shown in figures 1a and 1b). 

As per claims 17 and 18, the reasons and rationale for the rejection of claim 2 
are incorporated herein. Guell does not explicitly teach the claimed "detecting objects in 
the first fused image from the first type of sensor" and does not explicitly teach the 
claimed "combining the first fused image with symbols representing the objects". NASA 
teaches the claimed limitations by teaching of "Object Detection: Data was gathered to 
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help develop methods for detecting other aircraft" (third item in bulleted list under 
section "XVS Flight Test Series 11"). NASA teaches of this object detection being used 
with the first fused image from the first type of sensor because it differentiates "Object 
Detection" from "Object Detection via Radar" (object detection through the second fused 
image) in describing data collection methods in the bulleted list under section "XVS 
Flight Test Series II". 

As per claim 19, Guell teaches the claimed "transforming the first operator 
viewpoint image and the second operator viewpoint image to conform to the out-the- 
window visual scene" by teaching of "In essence, the actual outside world view can 
encompass a very large area that can go beyond what is visually allowed by the cockpit 
windows, providing unobstructed views in any desired direction (see figure 3)" (pg. 33, 
last sentence of 1st column, and top of second column). 

As per claims 20-23 and 41-45, Guell teaches the claimed "transforming the 
second fused image to the first operator viewpoint and to the second operator 
viewpoint" by teaching of "Each pilot has an independent view, which is controlled by 
their head position, while utilizing the same sensors that are static and fixed to the 
aircraft structure" (pg. 32, middle of first paragraph in first column) where the each 
pilot's independent view is an operator viewpoint. 
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Guell teaches the claimed "providing portions of the transformed image with data 
from a terrain map database" by teaching of a "Moving Map / RTIC" displayed as a 
portion of the transformed image in pg. 34 figure 6. 

Guell does not explicitly teach the claimed fusing the first and second images 
"with an enhanced image of a portion of the out-the-window scenery from at least one of 
the group of a RADAR sensor and a FLIR sensor, to generate a second fused image". 
Bernier teaches the claimed limitation by teaching of "Typical sensors used with the 
system are low-light video cameras, longwave infrared sensors, and millimeter wave 
radar, to name a few" (paragraph [0050]) where the long-wave infrared sensors perform 
a similar to function to the FLIR. 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to combine Bernier with the combinable system of Guell and NASA. One 
advantage of the combination is provided by Bernier, which teaches of "the system of 
the present invention may include two different types of sources ... the sources provide 
different images of the same field of view; each source having associated advantages 
and disadvantages. For example, one source could be a video camera . . . and the 
other source may be an infrared sensor that provides images based on heat sensing" 
(paragraph [0016]). The pilot of the system of Guell and NASA would benefit from 
image sources which provide image data in a variety of environmental conditions (i.e. 
low-visibility conditions) such as the infrared sources because it would help the pilot see 
more information as to the surroundings of the aircraft. 
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As per claims 25, 26, and 27, Guell does not explicitly teach the claimed "display 
processor is further operable to combine the viewpoint image with symbols, wherein the 
symbols represent information regarding the operational state of the device and the 
moving objects detected in the images", does not explicitly teach the claimed "display 
processor is further operable to detect moving objects in the first sensor image", and 
does not explicitly teach the claimed "display processor is further operable to generate 
symbols representing moving objects in the sensor image and the operational state of 
the device". NASA teaches the claimed limitations by teaching of tracking objects in the 
section titled "XVS Flight Test Series III" and by teaching of "pilot's ability to control and 
land the aircraft relying only on sensors and computer-generated images (including 
various symbols) on the XVS display" (3rd paragraph under the section titled "XVS 
Flight Test Series I"). NASA teaches of an advantage of displaying symbols of moving 
objects used in the XVS system by teaching of "The envisioned external Vision System 
(XVS) would guide pilots to an airport, warn them of other aircraft near their flight path, 
and provide additional visual aides for airport approaches, landings and takeoffsl" (2nd 
paragraph in article). NASA teaches of displaying symbols (a feature of the XVS 
system) that would warn pilots of other aircraft near their flight path. 

As per claim 28, this claim has limitations that follow those within claim 7 in terms 
of functionality, and thus are subject to the same reasons for rejection. 
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As per claim 29, this claim has limitations that follow those within claim 8 in terms 
of functionality, and thus are subject to the same reasons for rejection. 

As per claim 30, this claim has limitations that follow those within claim 9 in terms 
of functionality, and thus are subject to the same reasons for rejection. 

As per claim 31, this claim has limitations that follow those within claim 10 in 
terms of functionality, and thus are subject to the same reasons for rejection. 

As per claim 32, this claim has limitations that follow those within claim 1 1 in 
terms of functionality, and thus are subject to the same reasons for rejection. 

As per claims 33, 34, and 35, the reasons and rationale for the rejection of claims 
13, 14, and 20-23 are incorporated herein. Guell teaches the claimed "sensor is a video 
camera" by teaching of "video images from the cameras are digitized, and electronically 
combined in the Electronic Imaging System (EIS)" (pg. 33, first full paragraph in first 
column). 

Guell does not explicitly teach the claimed "second sensor is a RADAR sensor" 
and does not explicitly teach the claimed "second sensor is a FLIR sensor". Bernier 
teaches the claimed limitation by teaching of 'Typical sensors used with the system are 
low-light video cameras, long-wave infrared sensors, and millimeter wave radar, to 
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name a few" (paragraph [0050]) where the longwave infrared sensors perform a similar 
to function to the FLIR. 

As per claim 36, this claim has limitations that follow those within claim 1 in terms 
of functionality, and thus are subject to the same reasons for rejection. 

As per claim 38, this claim has limitations that follow those within claim 17 in 
terms of functionality, and thus are subject to the same reasons for rejection. 

As per claim 39, the reasons and rationale for the rejection of claims 2 and 18 
are incorporated herein. Guell does not explicitly teach the claimed limitation. Bernier 
teaches the claimed "primary flight information" by teaching of "For example, the 
aircrafts HSI and AD1 displays, altimeters, airspeed, etc, could be displayed on the 
display as a tile or fused with an image to provide an integrated view ... while also 
viewing the environment surrounding the aircraft" (paragraph [0102]). 

As per claims 46 and 47, the reasons and rationale for the rejection of claim 1 
are incorporated herein. Guell teaches the claimed limitations by teaching of "specific 
mission/operational needs" (pg. 33, top of 1st col) and by teaching of a "Control Panel" 
(figure 1 ). 
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Response to Arguments 

The objection to the title of the invention has been withdrawn in response to the 
new title proposed in this amendment 

The provisional double patenting rejection raised in the previous office action has 
been withdrawn in response to a terminal disclaimer filed on 1 1/10/2005 with copending 
application 10/616145. 

Applicant's arguments filed 11/10/2005 have been fully considered but they are 
not persuasive. 
Applicant argues: 

In rejecting claim 15, the examiner states that Bernier U.S. Patent Application 
Publication No' 2004/0169663 (hereafter "Bernier) shows multiple windows with a 
common display area that is customizable from alternative common display area views. 
(Office Action, page 15). With respect, Applicant disagrees. The cited portion of Bernier 
pertains to diagrams illustrating the results of percentage-based and content-based 
fusing of two images to form a composite image, as well as a diagram illustrating tiling 
of different images within the display. (Bernier, [0036] - [0038]). The processor receives 
separate images from separate sources and defines image meshes for each image. 
(Bernier [0095]). Further, the images overlap in large areas. Thus, Bernier does not 
disclose or suggest using two images that are mutually exclusive, nor are the images 
"windows of information on the display device" since they are not displayed separately. 

The examiner maintains that the prior art rejections in this regard are proper. 

Bernier in figure 16 shows a common window, 124 is generate from two separate image 

sources 122a and 122b (windows of information). Figure 18 of Bernier shows that the 

common window can have mutually exclusive (not overlapping) windows of information, 

134 and 136 and that the common window is customizable where the added windows of 

information 134 and 136 are customized added features. In addition, the prior art 
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rejections in regards to this limitation for this office action have been elaborated to better 
explain these teachings. 

Applicant argues the claimed "display devices positioned to provide the portion of 
a desired out-the window visual scene in combination with a window that provides 
another portion of the desired out-the-window visual scene" (claims 16, 24, and 37) is 
not taught in the cited section of the reference of Bernier as stated in the office action. 
Applicant further argues "In contrast, the cited portion of Bernier discloses integrating a 
view of the environment surrounding the aircraft with other types of data such as HIS, 
ADI, altimeters, airspeed, etc. (Bernier [0102]). The 'other' type of information in Bernier 
is not a window that provides another portion of the desired out-the-window visual 
scene". 

The examiner has elaborated the rejections in the above office action to address 

these arguments. Specifically, Bernier teaches the claimed limitation by teaching of 

"Importantly, also connected to the image processor is a display 28. In this particular 

embodiment, the display is a helmet-mounted display located in the helmet of the pilot 

or other viewer" (paragraph [0052]) and by teaching of "It is understood here that a 

helmet-tracking device is not required for the invention" (paragraph [0052]) where the 

reference would achieve the claimed limitations by using a monitor instead of a head 

mounted display (a different embodiment) where the monitor is placed in the cockpit 

(the cockpit with windows is shown in figures 1a and 1b). Bernier further suggests this 

limitation of using a monitor by teaching of: 

"Many modifications and other embodiments of the 
invention will come to mind to one skilled in the art to which 
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this invention pertains having the benefit of the teachings 
presented in the foregoing descriptions and the associated 
drawings. Therefore, it is to be understood that the invention 
is not to be limited to the specific embodiments disclosed 
and that modifications and other embodiments are intended 
to be included within the scope of the appended claims" 
(paragraph [0015]) 

and by teaching of: 

Obstructed vision is an important safety concern in 
aircraft navigation, and there has been considerable effort 
devoted to providing systems that increase or enhance a 
pilot's view from the cockpit, 
(paragraph [0006]) 

where such an action of using a monitor in the cockpit instead of a head-mounted 
display would constitute a modification obvious to one of ordinary skill in the art. One 
advantage to doing so may be to provide the operator with more freedom to operate the 
aircraft without wearing a helmet in which the operator may find to be uncomfortable or 
difficult to fit. 



Conclusion 

3. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
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mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .1 36(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Daniel F. Hajnik whose telephone number is (571 ) 272- 
7642. The examiner can normally be reached on Mon-Fri (8:30A-5:00P). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka J. Chauhan can be reached on (571) 272-7782. The fax phone 
number for the organization where this application or proceeding is assigned is 571- 
273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 





DFH 




